Title: Backups -- Revisited
Date: 2017-02-09 04:59
Category: backups
Tags: backups, CLI, borg, borgbackup, rsnapshot, sudoers.d, network, ncdu, gpg, awk, conky


###Backups
> blah blah blah.
> We all know, or should know, that backups are important.

As I mentioned in [my first post about backups], they are important.
I could link to so many statistics about lost data here... but then I would need to vet their legitimacy. \**wink*\*


###Options
Again, as I mentioned last time:
> There are a [Lot of Options] out there.
> A lot of them don't do what I want.
> There are a few criteria that I desire, deltas, encrypted transfer, encrypted storage, space conservation, remote backup.
> There are few programs that did these, and did them in a way I liked.

Last time I talked about `rsnapshot` and `obnam`.
After close to a year of using my [cryptshotr] (a fork of [cryptshot]), a `rsnapshot` wrapper, I have learned more about what I want.
I have also gotten myself a shiny NAS recently.


####NAS
I got a QNAP TS-453A recently, and put four 4TB WesternDigital Red Drives in it.
Sadly, one drive was dead on arrival, and I had to go through warranty.

After getting the replacement drive, I now have a RAID6 with ~7.1TB of usable space!
<sup>(About two terabytes were immediately filled with content)</sup>

Having gotten this nice, big, directly networked, high network speed/disk speed beast, I decided it was time to relook at my backups.


####rsnapshot
My [rsnapshot] based backups have worked, and were reliable, but had two 'issues'.
They where slower than I would like. An incremental backup would take about 15-20 minutes.
And they use an encrypted `LUKS`+`dmcrypt`, which isn't going to fly with my NAS.

Now, I'm sure that a big part of the time sink was needing to decrypt/mount the USB 3 external for every backup. <sup>(And the pi3 only has so much CPU...)</sup>

But to be honest, `cryptshotr` is also a bit [Rube Goldberg-esk].
![cryptshotr diagram]({static}/pics/backups-cryptshotr.png)


####Attic
After yet another round of backup option research, [Attic] was looking like a decent option.

* works over the network
* has built in encryption
* does deltas
* does de-duplication

However when I went to install in on debian sid, I found it was no longer in the repo.
With a bit more digging, I notices it was looking a bit *stale*.
While looking at the git issues, I noticed [ThomasWaldmann] referencing issues to [BorgBackup], so I gave it a peek.


###BorgBackup (aka borg)
####[What are the differences between Attic and Borg?]
> Borg is a fork of Attic and maintained by “The Borg collective”.

> Here’s a (incomplete) list of some major changes:

> * more open, faster paced development (see issue #1)
> * lots of attic issues fixed (see issue #5)
> * less chunk management overhead (less memory and disk usage for chunks index)
> * faster remote cache resync (useful when backing up multiple machines into same repo)
> * compression: no, lz4, zlib or lzma compression, adjustable compression levels
> * repokey replaces problematic passphrase mode (you can’t change the passphrase nor the pbkdf2 iteration count in “passphrase” mode)
> * simple sparse file support, great for virtual machine disk files
> * can read special files (e.g. block devices) or from stdin, write to stdout
> * mkdir-based locking is more compatible than attic’s posix locking
> * uses fadvise to not spoil / blow up the fs cache
> * better error messages / exception handling
> * better logging, screen output, progress indication
> * tested on misc. Linux systems, 32 and 64bit, FreeBSD, OpenBSD, NetBSD, Mac OS X


####Usage
Usage is pretty simple. To initialize a backup location `borg init /path/to/repo` and to make a backup `borg create /path/to/repo::archive_name_x ~/one ~/two`.
Then you can start dropping options on top of that.


####AutoFS
Initially I was pointing my backups at an [AutoFS] local `CIFS` mount, which worked quite well.
\**skims past*\*
After getting everything working and automated <sup>(more on that later)</sup>, I decided to switch over to the ssh connection type so I could start backing up my VPS too.<sup>(Since... uh... I'm not going to be doing a CIFS mount over the Internet)</sup>

***I should also note that incremental updates are only taking one to three minutes now!!! (with ~45gb of space that `borg` is paying attention to).***


###Setting up SSH
This proved to be a big of a chore.
With my QNAP I have a third party package called [Entware-ng] install, which provides the `opkg` package manager from [OpenWRT] (and I have used on my router).
This is a handy way to get various linux packages onto the NAS, like my `deluged`.
Unfortunately `borg` wasn't an offered package.
The precompiled binaries also have [issues].


####QNAP Container
This seemed pretty easy.
Install the app called 'Container Station', and wait for it to install itself and its dependency apps.
Then make a container of your chosen linux distro <sup>(please don't say 'ubuntu'...)</sup>.
I choose a debian lxc <sup>(which I promptly upgraded to debian sid)</sup>.
During the setup process, select 'Advanced Settings>>>', set up the Port Forwarding, and the Shared Folders. <sup>(I didn't do this the first time round...)</sup>


#####Fixing QNAP Container Port Forwarding
So... about Port Forwarding...
The UI only lets you edit it during container creation...

Bug anyone?

At this point I start using my GoogleFu, and see that a few people have posted about this over the past couple of years... with no solutions known.

Oooo the bug gets funner! After restarting my container, the Port Forward vanished!

Well, I decide it can't hurt to start digging though the files on the filesystem, and I luck out.
Behold the `qnap.json`, found at `/share/ROOT_OF_RAID/SHARE_NAME/container-station-data/lib/lxc/NAME_OF_CONTAINER`.
In my case: `/share/CACHEDEV1_DATA/containers/container-station-data/lib/lxc/debian`.

Reading the now *'bad'* file, I find:

    :::php
    {"volume": {"new": [], "host": {}, "container": []}, "version": "8", "resource": {"device": [], "limit": {}}, "name": "bad-container", "autostart": true, "arch": "amd64", "image": "debian-jessie", "type": "lxc", "network": {"hostname": "bad-container", "port": [], "mode": "nat"}}

After making a through away container, I find a good one should look like:

    :::php
    {"volume": {"new": [], "host": {}, "container": []}, "version": "8", "resource": {"device": [], "limit": {}}, "name": "throw-away", "autostart": true, "arch": "amd64", "image": "debian-jessie", "type": "lxc", "network": {"hostname": "throw-away", "port": [[1111, 22, "TCP"]], "mode": "nat"}}

Where `[[1111, 22, "TCP"]]` is `[[Exterior_port, container_port, "TCP_or_UDP"]]`.
While I did not test, I suspect multiple ports would look like `[[1111, 22, "TCP"], [2222, 8080, "TCP"], [ETC...]]`

Which is now [posted] on the QNAP forum as well.


###End Setup
Here is a summary of how I tied everything together.


####borg_wrap.sh
I wrote up a nice little script to handle my backups.

It features:

* Error handling
* Config items, and config checking
* Tests that I am on home network
* `pass` ([passwordstore]) is used to store the password
    * Relies on gpg key being cached to access password

[borg_wrap.sh code]


####Users
Users? We don't need this section!
Well, aside from a normal account on the destination with access to the backup destination.


####SSH
    :::bash
    ssh-ed25519 XXX... laptop_key
    command="borg serve --restrict-to-path /mnt/borg/laptop",no-pty,no-agent-forwarding,no-port-forwarding,no-X11-forwarding,no-user-rc ssh-ed25519 XXX...
    command="borg serve --restrict-to-path /mnt/borg/server",no-pty,no-agent-forwarding,no-port-forwarding,no-X11-forwarding,no-user-rc ssh-ed25519 XXX...

Notable parts:

* The script calls a passphraseless key, so that it doesn't need to source ssh-agent.
    * Trying to source ssh-agent with crontab is possible, but sudo just makes that a painful thought.
* The `borg serve --restirct-to-path` both limits borg serve's access, and prevents non-matching commands from working -- thus preventing unauthorized access.
* Due to QNAP's... Interesting Port Forwarding set up, I can't limit the keys with `from=`, as it isn't seen correctly by the container.


####Sudoers.d
Here is the content of my `/etc/sudoers.d/borg_sudoers`:

    :::bash
    demure ALL=(ALL) NOPASSWD: /usr/local/sbin/borg_wrap.sh -q, /usr/local/sbin/borg_wrap.sh

Compared to my old `cryptshotr` it is very simple and concise.

Notable parts:

* `borg_wrap.sh` is moved to `/usr/local/sbin` to prevent edits for user escalation.
* `/etc/sudoers.d` is used to prevent issues with bad edits breaking `sudo` completely, and on some OS's to survive `sudo` upgrades.
* NOPASSWD is used so that it can run automated.


####crontab
I have `borg_wrap.sh` called every hour, and unlike my old `cryptshotr` I don't bother with fancy time checks as I don't need to balance/track multiple periodicities.

    :::bash
    ## borgbackup wrapper
    0 */1 * * * sudo /usr/local/sbin/borg_wrap.sh -q

**Note**: Aside from how often you run backups, periodicities are partially a function of how you prune.


#####Pruning
While I have not accumulated enough backups to use this yet, this is close to what I will be using:

    :::bash
    borg prune -v --list --dry-run -d=7 -w=4 -m=12 -y=-1 --keep-within=1d /path/to/repo

Which will keep:

* all archives <= 24hours old
* one archive per day <= 7days old
* one archive per week <= 4weeks old
* one archive per month <= 12months old
* unlimited yearly archives


####conky
As a finishing touch, I updated my `back_log_backups.conkyrc` to use my `borg_wrap.sh` log, stored at `$HOME/.config/borg/log`.
This will show the last six entries, and if none six are a 'passed' it will show a seventh line at the top, showing the last 'passed' entry.
![borg conky]({static}/pics/borg-conky.png)

[back_log_backups.conkyr code]


###borg_wrap.sh Diagram
So, as you can see, my setup is much more simplified now:
![borg_wrap.sh diagram]({static}/pics/backups-borg_wrap.png)

<sup>**2017JUL09:** Updated picture as I noticed an error.</sup>


###Afterword
Having done all the leg work for my laptop, it was easy to add another repo/authorized_key for my VPS and just tweak the script (and disable the wifi MAC address lookup).
So now I have both my laptop and my VPS backed up on my NAS.


###Links
[my first post about backups]  
[Lot of Options]  
[rsnapshot]  
[cryptshotr]  
[cryptshot]  
[ThomasWaldmann]  
[BorgBackup]  
[Rube Goldberg-esk]  
[Attic]  
[What are the differences between Attic and Borg?]  
[AutoFS]  
[Entware-ng]  
[OpenWRT]  
[issues]  
[posted]  
[passwordstore]  
[borg_wrap.sh code]  
[back_log_backups.conkyr code]  


[my first post about backups]: {static}008-backups_conky_display.md
[Lot of Options]: https://wiki.archlinux.org/index.php/Synchronization_and_backup_programs
[rsnapshot]: http://rsnapshot.org/
[cryptshotr]: https://notabug.org/demure/cryptshotr/src/master/cryptshotr
[cryptshot]: https://github.com/pigmonkey/backups/blob/master/cryptshot.sh
[ThomasWaldmann]: https://twitter.com/thomasjwaldmann?lang=en
[BorgBackup]: https://github.com/borgbackup/borg
[Rube Goldberg-esk]: https://en.wikipedia.org/wiki/Rube_Goldberg_machine
[Attic]: https://github.com/jborg/attic
[What are the differences between Attic and Borg?]: https://borgbackup.readthedocs.io/en/stable/faq.html#what-are-the-differences-between-attic-and-borg
[AutoFS]: https://wiki.archlinux.org/index.php/autofs
[Entware-ng]: https://github.com/Entware-ng/Entware-ng
[OpenWRT]: https://openwrt.org/
[issues]: https://github.com/borgbackup/borg/issues/690
[posted]: https://forum.qnap.com/viewtopic.php?t=115258#p596749
[passwordstore]: https://www.passwordstore.org/
[borg_wrap.sh code]: https://notabug.org/demure/scripts/src/master/borg_wrap.sh
[back_log_backups.conkyr code]: https://notabug.org/demure/dotfiles/src/master/i3/conky/back_backups.conkyrc

